The invention pertains to electronic artificial neural networks, in which many simple electronic units are heavily interconnected such that they are capable of massively parallel computation. As is customary in the discussion of artificial neural networks, functional parts are given names suggested by neurobiology, and the adjective "artificial" is suppressed; hence we will speak here simply of "neurons", "synapses", and "dendrites".
With this understanding, a neural net may be described as a collection of functional units, called neurons, which are interconnected via junctions, called synapses, and input lines called dendrites. Each dendrite collects synaptic outputs into a sum, called activation, which is presented to a neuron for processing according to an output function, thereby producing a signal. In electronic neural nets, the signals are usually voltages, the activations are electric currents, and the neurons are operational amplifiers which have an activation current as input, and a signal voltage as output. The output function is usually of sigmoidal type, such as the hyperbolic tangent. The dendrite of the neuron is a conductor that collects the output currents of certain synapses. The synaptic inputs are signals from certain neurons.
Neural networks can be structured in such a manner that the net functions as an associative memory, i.e., a device that is capable of associative recall, in which an input vector x produces as output the stored vector that is closest to x. Undesirable consequences of correlations between stored vectors can be eliminated by encoding the stored vectors as orthogonal bipolar vectors, i.e., orthogonal vectors with components 1 and -1. Such vectors are called Hadamard vectors. Encoding the stored vectors q.sub..alpha., .alpha.=1 to L, as Hadamard vectors means that to every stored vector q.sub..alpha. is assigned a Hadamard vector h.sub..alpha.. These assignments may be expressed in an outer products matrix ##EQU1## where T denotes transposition.
The encoding is used in the following manner. From an input vector x one forms the vector ##EQU2## this is just a linear combination of Hadamard vectors h.sub..alpha., with coefficients c.sub..alpha. =q.sub..alpha. x.sup.T, the scalar products of x with the vectors q.sub..alpha.. The operation Bx.sup.T is performed by a device called Bidirectional Linear Transformer (BLT). The bidirectional feature will become clear presently. The BLT is followed by a device that selects, from the linear combination ##EQU3## the Hadamard vector that goes with the largest of the coefficients c.sub..alpha., if unique. Hence, if c.sub..beta. &gt;c.sub..alpha. for all .alpha..noteq..beta., then the Hadamard vector h.sub..beta. is selected. The device has been named Dominant Label Selector (DLS), because the code vectors h.sub..alpha. serve as labels for the stored vectors q.sub..alpha.. The output h.sub..beta. of the DLS is returned to the BLT for a backstroke, in which the BLT produces the vector ##EQU4## where N is the dimension of the Hadamard vectors. The result holds because the Hadamard vectors are orthogonal and have the Euclidean norm .sqroot.N. Division by N or, in case of a bipolar x, thresholding with a sigmoidal function that ranges from -1 to 1, gives x'=q.sub..beta.. But .beta. is the index for which c.sub..alpha. is maximum. Since c.sub..alpha. =q.sub..alpha. x.sup.T, the vector q.sub..beta. is the stored vector which has the largest scalar product with the input vector x. Therefore, if the DLS indeed selects from u=Bx.sup.T the dominant Hadamard vector, then the device consisting of the BLT and the DLS has perfect associative recall of up to N stored vectors. The device has been called Selective Reflexive Memory (SRM) [1-4]. Its front end, the BLT, may be seen as a Bidirectional Associative Memory (BAM) [5-7], with the rear thresholding removed, and the front thresholding optional. The DLS in the rear may be seen as a distributed winner-take-all net; instead of selecting from an analog vector the maximium component, the net selects the largest term in the Hadamard expansion of the vector. Distributing the winner-take-all process improves fault tolerance, since there are then no grandmother cells.
Since the DLS output is the Hadamard vector that is closest to the vector u, the DLS may itself be seen as an associative memory with a complete set of Hadamard vectors as stored states. Therefore, this DLS is called a Hadamard memory.
A Hadamard memory cannot be constructed with the customary linear activation. Instead, one may consider a net with quadratic activation, so that the total current in the dendrite of neuron i is ##EQU5## where y.sub.i is the signal from neuron i, determined from the activation v.sub.i of the neuron by the output function s(.), EQU y.sub.i =s(v.sub.i). (2)
The function s(.) is restricted to be sigmoidal and antisymmetric, and to have the range [-1,1]. Hence, fully developed neuron signals are bipolar. All indices range from 1 to N, restricted to be a power of 2. In the simplest continuum model, the equations of motion are EQU v.sub.i =-v.sub.i +I.sub.i +r.sub.i, (3)
where dot denotes differentiation with respect to time, and the term r.sub.i expresses a threshold or an external coupling. The first two terms in (3) have unit coefficients, but that does not constitute a physical restriction, since this form can always be obtained by scaling of the time and activation, together with a related adjustment of the neuron output function s(.). In view of the quadratic form of the dendrite current (1), one has here a case of higher-order neurons [8,9]. A Hadamard memory is obtained by choosing the connection tensor as ##EQU6## where h.sub..alpha.i is the ith component of the Hadamard vector h.sub..alpha.. It has been shown [8] that a neural net with nonlinear activation is stable if the connection tensor is symmetric, and if all tensor components for which two or more indices are equal vanish. The connection tensor given by (4) is indeed symmetric. In order to satisfy the second condition, subtractions have to be applied, to give the tensor ##EQU7## where .delta..sub.ij is the Kronecker delta. The subtractions are correct for a choice of Hadamard vectors such that their first component is +1, and the vector h.sub.1 has all components +1. The connection tensor (5) is referred to as subtracted. It can be shown that all nonzero connection tensor components have the same value, which is positive; this is true for (4) as well as for (5). Hence, in a Hadamard memory, all synapses are excitatory, and they all have the same strength. Up to couplings, the structure of the memory is entirely determined by the connections.
The BLT output must be coupled to the Hadamard memory. This may be done in several ways. In initial value coupling, the BLT output u is used as an initial value of the activation v, after multiplying with a properly chosen coupling constant. The term r.sub.i in the equation of motion (3) is then taken as constant, usually zero. In the external coupling the activation is started out at zero, and the BLT output u.sub.i is applied to the term r.sub.i of (3), after multiplication with a coupling constant. Combined coupling involves a combination of these schemes.
Computer simulations have shown the Hadamard memory to have perfect associative recall, for N=8 and 16, for each of these couplings, for a range of coupling constants, and for unsubtracted as well as subtracted connection tensors, (4) and (5).
In practice, Hadamard memories become particularly important for large dimension N. For N a power of 2, and with well-chosen Hadamard vectors, the number of product synapses is N(N-1)/2 and (N-1)(N-2)/2 respectively for unsubtracted and subtracted connection tensors; these numbers of connections are about the same as for a fully connected Hopfield memory [10]. It is difficult to construct so many connections in electronic implementations for large dimension N. For instance, for N=1024, one would need 522753 connections in the subtracted case. Furthermore, each synapse must compute the product of its two inputs, and that requires a four-quadrant multiplier, with at least 9 transistors [11]. For the example mentioned above, that comes to 4.7 million transistors for the synapses alone! It is an object of the present invention to overcome the problem of the large number of connections and synapse transistors, for Hadamard memories of large dimension N.
With the Hadamard memory used as the rear stage of an SRM, many applications require the BLT to be adaptive. The connection matrix ##EQU8## of the BLT has outer-products structure, and it can therefore be modified by Hebb learning [12]. This requires that a new Hadamard vector be presented to the back of the BLT, whenever a new vector q.sub..alpha. is to be stored. An extension to the Hadamard memory is needed, that causes the memory to provide such vectors in a simple manner, when learning is required. It is the further object of the present invention to furnish such an extension.